Scalable Game Primitives / Distributed Real-Time Aggregation Of Player Data

ABSTRACT

Disclosed herein is a set of coordinated game service primitives that allows for custom, per-game, logic. A new game with new custom logic can be added without needing to change any public api or database schema or write any new server code. The same hardware may be shared across many games. And the system scales linearly. That is, if the number of servers is doubled, then so is the number of concurrent users that can be supported. Also disclosed are systems and methods for two-stage aggregation. Pre-aggregation may take place on a plurality of front-end server. Final aggregation may take place on one or more back-end servers. A name value store may be used to reliably store state.

BACKGROUND

In a console game designed to support many users all posting messages toa server or servers, where each message contains data that needs to beaggregated and made available back to those users, common “primitive”aggregation logic, such as summation and top-n, as well as customgame-specific aggregation logic can be used to drive custom game logic.For example, each user might belong to a team and post a score. Thescores might need to be summed to get the total score for each team. Theaggregate score might need to be made available quickly to the users.Writing custom server code is expensive, however, and custom servers areexpensive to maintain.

Also, in such a console game, due to the volume of posted messages, allthe posts may not be able to be handled on a single front end server.Accordingly, it would be desirable to provide a mechanism foraggregating values posted across many front end servers.

In a platform that supports many massively multi-player games, whichtend to have a high number of concurrent users during just a few hours aweek, many servers may be needed to support peak load, which may be onthe order of about 200,000 internet accesses/second, with peak datatransfer rates on the order of Gigabits/second. For this reason andothers, it may be desirable to share a single game service across manydifferent games.

It would be particularly desirable if a single infrastructure thatscales to peak could be used across many games, with a standard servicethat need not be extended every time a new game is built. Such a serviceshould be sufficiently fault tolerant such that it can tolerate andrecover from any single component failure automatically and withinseconds.

SUMMARY

Disclosed herein is a set of coordinated game service primitives thatallows for custom, per-game, logic. The functionality of the primitivesdoes not change from game-to-game, custom, client-side game logic can bebuilt on top of these primitives in much the same way that customreports can be generated using SQL primitives. For example, a new gamewith new custom logic can be added without needing to change any publicapplication programming interface (API) or database schema, nor to writeany new service or server-side code. The same hardware may be sharedacross many games. And the system scales linearly. That is, if thenumber of servers is doubled, then so is the number of concurrent usersthat can be supported.

Also disclosed are systems and methods for two-stage aggregation.Pre-aggregation may take place on a plurality of front-end servers.Final aggregation may take place on one or more back-end servers. A namevalue store may be used to reliably store state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example network-based game service.

FIG. 2 is a flowchart for employing scalable game service primitives ina network-based game service.

FIG. 3 is a block diagram of an example computing environment in whichexample embodiments and aspects of the present invention may beimplemented.

FIG. 4 is an example network configuration in which aspects of theinvention may be implemented.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

As shown in FIG. 1, an example network-based game service may include aplurality of front-end servers, one or more back-end servers, and apartitioned name/value store.

The front-end servers may be TCP/IP Socket Servers that dynamically loadthe pre-aggregation primitive code and custom game logic that may bereferred to as “front-end controllers.” These front-end controllersprocess incoming messages from users and perform custom game logic onthem, often storing the results in the name/value store. Front-Endcontrollers may read and write from the name/value store.

The back-end servers similarly load back-end controllers. Back-endcontrollers do the final aggregation of pre-aggregated data (e.g.,aggregating a sum for some data pre-aggregated and set by a plurality offront end servers) and performing other custom game logic includingadvancing a round. Back-end controllers read and write from thename/value store.

The name/value store is a partitioned sql database with a simple,single-table schema. The name/value table has two columns, one for nameand one for value. The name is a string and the value is a binary blob,which may be an xml blob, for example. The store is partitioned on name.The name/value store does not provide transactions or record locking.

As shown in FIG. 2, users may post messages to the front end controllersat 52. The messages, each of which may have a name and an XML body, arepre-aggregated on the front-end servers.

At 54, messages are routed by name to the appropriate front endcontrollers.

At 56, the front end controllers execute on all the messages in a batch,often aggregating and writing the aggregate value to the name/valuestore.

At 58, the back-end servers run their controllers, periodically readingvalues from the name/value store and perhaps further aggregating theresults and writing out the final values. So that the back end serversknow which values to aggregate, a queuing mechanism may be employed tosignal the back end server. The queuing mechanism may be polledperiodically.

Consider an example of 200,000 game consoles playing a quiz-show gameall at the same time. Each game console may connect to the data centerthrough a secure gateway, and communicate using encrypted traffic. Thegame consoles may tunnel through the secure gateway to the plurality offront-end servers. The game consoles may connect using persistent TCP/IPconnections rather than web service calls. To handle 200,000 gameconsoles, the system may include, say, 20 front-end servers, each tohandle 10,000 game console clients.

Suppose that each of the 200,000 game consoles is answering the samequestion at the same time. Each game console submits its respectiveanswer, and each front end gets 10,000 messages at same time.

Typically, each user will want to know whether he or she got the rightanswer. But the users might also want feedback that relies on answersprovided by other users. For example, the users might want to know whatpercentage of respondents got the right answer, or who was first toanswer correctly.

The individual game consoles may be programmed to determine whether theuser got an answer right or wrong, and provide such an indication to theuser. Each game console may keep score of how many answers the user ofthat console got right. Each game console may then communicate to theassociated front-end server whether the user got the answer right orwrong. Data may then be aggregated on the back-end server.

Data is tracked in the system in accordance with name/value pairs. Thatis, every datum may have a name and a value associated with it. Anexample of a name might be game title, such as“trivia.quest.sessionid.questionno.right/wrong.” Values for this datummight be “right” and “wrong,” which could be represented as “0” and “1,”respectively.

Each front-end server does pre-aggregation (e.g., a summation for theirsubset of users), and writes the pre-aggregated value (e.g., a sum) outto the store. This process may be referred to as “pre-aggregation,”i.e., aggregation by the front-end server of input data provided by eachof a plurality of clients associated with the front-end server. Theback-end server grabs the list of sums and sums the sums to get a finalsum of sums. Thus, the back-end server aggregates the data globally.

To accomplish this, the back-end servers need understand only a set ofbasic primitives. The console sends a name and value, and the back-endserver sums all the values over that name. For example, a console mightsend a message with a name such as “question 7 right?” and a value of“1” or “0” (depending on whether the user got it right or not). Theaction (i.e., primitive) may be “sum.” After a prescribed period of waittime (e.g., six seconds), the game console may ask the front-end serverto report back an aggregation from the universe of consoles. Forexample, the game console may send a message with name =“question 7right?” and action (primitive)=“get.” In response, the back-end servermay provide the aggregate value for that name (which, in this example,is the summation of the number right).

Note that the front-end server need not communicate directly with theback-end server. The front-end server may communicate values to thename/value store. The back-end server may pick up pre-aggregated valuesfrom the name/value store, and write aggregated values back. Theback-end server may pick up aggregated values from the name/value store.

Example Computing Environment

FIG. 3 shows an exemplary computing environment in which exampleembodiments and aspects may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality. Neither should the computing environment 100 beinterpreted as having any dependency or requirement relating to any oneor combination of components illustrated in the exemplary operatingenvironment 100.

Numerous other general purpose or special purpose computing systemenvironments or configurations may be used. Examples of well knowncomputing systems, environments, and/or configurations that may besuitable for use include, but are not limited to, personal computers,server computers, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputers, mainframe computers, embeddedsystems, distributed computing environments that include any of theabove systems or devices, and the like.

Computer-executable instructions, such as program modules, beingexecuted by a computer may be used. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Distributed computing environments may be used where tasks are performedby remote processing devices that are linked through a communicationsnetwork or other data transmission medium. In a distributed computingenvironment, program modules and other data may be located in both localand remote computer storage media including memory storage devices.

With reference to FIG. 3, an exemplary system includes a general purposecomputing device in the form of a computer 110. Components of computer110 may include, but are not limited to, a processing unit 120, a systemmemory 130, and a system bus 121 that couples various system componentsincluding the system memory to the processing unit 120. The processingunit 120 may represent multiple logical processing units such as thosesupported on a multi-threaded processor. The system bus 121 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. By way of example, and not limitation, sucharchitectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus (also known as Mezzanine bus). Thesystem bus 121 may also be implemented as a point-to-point connection,switching fabric, or the like, among the communicating devices.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CDROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 3 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 3 illustrates a hard disk drive 140 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156, such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 3, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 3, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 20 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the systembus, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 3. The logical connections depicted in FIG. 3include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 3 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

Network Environment

FIG. 4 illustrates an example network environment in which the presentinvention may be employed. Of course, actual network and databaseenvironments may be arranged in a variety of configurations; however,the example environment shown here provides a framework forunderstanding the type of environment in which an embodiment mayoperate.

The example network may include one or more client computers 200 a, aserver computer 200 b, data source computers 200 c, and/or databases270, 272 a, and 272 b. The client computers 200 a and the data sourcecomputers 200 c may be in electronic communication with the servercomputer 200 b by way of the communications network 280 (e.g., anintranet, the Internet or the like). The client computers 200 a and datasource computers 200 c may be connected to the communications network byway of communications interfaces 282. The communications interfaces 282can be any type of communications interfaces such as Ethernetconnections, modem connections, wireless connections and so on.

The server computer 200 b may provide management of the database 270 byway of database server system software such as MICROSOFT®'s SQL SERVERor the like. As such, server 200 b may act as a storehouse of data froma variety of data sources and provides that data to a variety of dataconsumers.

In the example network environment of FIG. 4, a data source may beprovided by data source computer 200 c. Data source computer 200 c maycommunicate data to server computer 200 b via communications network280, which may be a LAN, WAN, Intranet, Internet, or the like. Datasource computer 200 c may store data locally in database 272 a, whichmay be database server or the like. The data provided by data source 200c can be combined and stored in a large database such as a datawarehouse maintained by server 200 b.

Client computers 200 a that desire to use the data stored by servercomputer 200 b can access the database 270 via communications network280. Client computers 200 a access the data by way of, for example, aquery, a form, etc. It will be appreciated that any configuration ofcomputers is equally compatible with an embodiment of the presentinvention.

1. A network-based gaming system, comprising: a plurality of front-endservers, each said front-end server executing thereon respectivefront-end controllers; a back-end server; and a partitioned name/valuestore, wherein users are enabled to post messages to the front-endcontrollers, each said message having a respective name and body,messages are routed by name to the appropriate front-end controllers,the front-end controllers aggregate values from a batch of messages, andwrite the aggregated values to a persistent data store, and wherein theback-end servers periodically read aggregated values from the datastore, and write to the data store final values aggregated over theplurality of front-end servers.
 2. The system of claim 1, wherein thefront-end servers perform primitive logic on the incoming messages,store results in the data store, read from the data store, and supplyaggregated results back to the users.
 3. The system of claim 1, whereinthe front-end servers are adapted to process incoming messages fromusers, to perform custom game logic on the incoming messages, to storeresults in the name/value store, and to read from the name/value store.4. The system of claim 1, wherein the back-end servers are adapted toperform time-based operations and other custom game logic, and to readand write from the name/value store.
 5. The system of claim 1, whereinthe data store is a partitioned name/value store with a single-tableschema.
 6. The system of claim 5, wherein the single table is aname/value table having a first column for name and a second column forvalue.
 7. In network-based gaming system comprising a plurality offront-end servers, each front-end server being associated with arespective plurality of game consoles, a single back-end serverassociated with the plurality of front-end servers, and a partitionedname/value store, a method comprising: receiving at each of thefront-end servers, a set of messages from the respective plurality ofgame consoles associated therewith, each said message having arespective name and XML body; aggregating at each of the front-endservers values from the received messages; writing the aggregated valuesfrom the front-end servers to the name/value store; periodically readingthe aggregated values at the back-end server from the name/value store;and writing from the back-end server to the name/value store a finalvalue aggregated over the plurality of front-end servers.
 8. The methodof claim 7, further comprising: routing the messages by name to thefront-end controllers.
 9. The method of claim 7, further wherein data istracked in the system in accordance with name/value pairs.
 10. Themethod of claim 7, wherein each front-end server performspre-aggregation of input data provided by each of the plurality ofclients associated with the front-end server.
 11. The method of claim10, wherein the back-end server aggregates the data globally.
 12. Themethod of claim 11, wherein the aggregation is performed using a set ofbasic primitives.
 13. A network-based gaming system, comprising: aplurality of front-end servers, each said front-end server executingthereon respective front-end controllers; a back-end server; and apartitioned name/value store, wherein each front-end server performspre-aggregation of input data provided by each of a plurality of clientsassociated with the front-end server, wherein the back-end serveraggregates the data globally, and wherein the aggregation is performedusing a set of basic primitives.
 14. The system of claim 13, whereinusers are enabled to post messages to the front-end servers, each saidmessage having a respective name and XML body.
 15. The system of claim14, wherein each message contains a respective value associated withsaid name, and the back-end server aggregates the data by summing thevalues over a set of messages containing the name.
 16. The system ofclaim 13, wherein the set of primitives includes at least get and setprimitives.
 17. The system of claim 13, wherein the set of primitivesincludes at least sum and get primitives.
 18. The system of claim 13,wherein the front-end server communicates pre-aggregated values to thename/value store, and does not communicate directly with the back-endserver.
 19. The system of claim 18, wherein the back-end server picks uppre-aggregated values from the name/value store, and writes aggregatedvalues back to the name/value store.
 20. The system of claim 19, whereinthe back-end server picks up the aggregated values from the name/valuestore for communication to the users.